The Optimization Landscape for Fitting a Rank-2 Tensor with a Rank-2 Tensor∗

The ability to approximate a multivariate function/tensor as a sum of separable functions/tensors is quite useful, but algorithms to do so exhibit unpleasantly interesting behavior known as swamps. Such swamps have been the bane of users for decades and need to be understood so they can be alleviated. In previous work, we developed and applied dynamical systems concepts to analyze the simplest nontrivial case of tensor approximation, which is a rank-1 tensor approximating a rank-2 tensor. Now we consider the next, perhaps most important, case of a rank-2 tensor approximating a rank-2 tensor. We find terminal swamps caused by ill-conditioned Hessians at the solution, and that these worsen for small angle; this mechanism is consistent with the literature and not surprising. We find transient swamps caused by descent along valleys emanating from essential singularities; this mechanism is surprising and not amenable to standard analysis, which explains why it has been so hard to identify.